Exploring Self-Distillation Based Relational Reasoning Training for Document-Level Relation Extraction

Document-level relation extraction (RE) aims to extract relational triples from a document. One of its primary challenges is predict implicit relations between entities, which are not explicitly expressed in the document but can usually be extracted through reasoning. Previous methods mainly implicitly model reasoning interaction among entities or entity pairs. However, they suffer two deficiencies: 1) often consider only one pattern, coverage on limited; 2) do process In this paper, deal with first problem, we propose document-level RE module that contains core unit, multi-head self-attention unit. This unit variant conventional and utilizes four attention heads common patterns, respectively, cover more than previous methods. Then, address second issue, self-distillation training framework, branches sharing parameters. branch, randomly mask some pair feature vectors document, then train our infer their by exploiting information other related By doing so, because additional masking operation used during testing, it causes an input gap testing scenarios, would hurt performance. To reduce gap, perform supervised without branch utilize Kullback-Leibler divergence loss minimize difference predictions branches. Finally, conduct comprehensive experiments three benchmark datasets, experimental results demonstrate consistently outperforms all competitive baselines. Our source code available at https://github.com/DeepLearnXMU/DocRE-SD